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Abstract 

Computational fluid dynamics (CFD) is necessary in the design processes of all current 
aerospace plane programs. Single-stage-to-orbit (SSTO) aerospace planes with air-breathing 
supersonic combustion are going to be largely designed by means of CFD. The challenge of 
the aerospace plane design is to provide credible CFD results to work from, to assess the 
risk associated with the use of those results, and to certify CFD codes that produce credible 
results. To establish the credibility of CFD results used in design, the following topics 
are discussed: CFD validation vis-a-vis “measurable” fluid dynamics (MFD) validation; 
responsibility for credibility; credibility requirement; and a guide for establishing credibility. 
Quantification of CFD uncertainties helps to assess success risk and safety risks, and the 
development of CFD as a design tool requires code certification. This challenge is managed 
by “designing” the designers to use CFD effectively, by ensuring quality control, and by 
balancing the design process. For designing the designers, the following topics are discussed: 
how CFD design technology is developed; the reasons Japanese companies, by and large, 
produce goods of higher quality than their U.S. counterparts; teamwork as a new way of 
doing business; and how ideas, quality, and teaming can be brought together. Quality 
control for reducing the loss imparted to the society begins with the quality of the CFD 
results used in the design process, and balancing the design process means using a judicious 
balance of CFD and MFD. 

Introduction 

During the 1980s aerospace plane programs were begun in every major technological 
nation. Unmanned and manned, and single-stage-to-orbit (SSTO) and two-stages-to-orbit 
(TSTO) designs are being pursued. SSTO designs are fully reusable, whereas TSTO designs 
may be either fully reusable or semi-reusable. Some of the aerospace plane programs call 
for building experimental planes, some for operational planes. Most of these programs are 
using largely state-of-the-art technologies (e. g. see Ref. 1). However, programs undertaken 
to design SSTO aerospace planes with supersonic combustion require critical new technolo- 
gies, principally that of air-breathing propulsion over a wide range of Mach numbers. A 
significant and common factor among all aerospace plane programs is computational fluid 
dynamics (CFD), which is essential to the design process. (CFD is used herein to encompass 
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a range of related areas, in which computations are done, such as aerodynamics, combustion, 
rarified gas dynamics, and aerothermodynamics.) The objective of this paper is to define 
the challenge that designing aerospace planes presents to the computational fluid dynamics 
community and to present a plan for addressing this challenge. 

The design process for a new fluid dynamics system can begin only when there is at 
least some understanding of the flow field associated with that system. The requirements 
and technologies related to materials, structures, control systems, etc. are determined only 
after some information about the flow field is acquired. For example, it is the flow field 
that deter mi nes structural heat loads which in turn determine the characteristics of the 
materials required. If such materials are not available, then the necessary technology needs 
to be developed in order to produce them. Once the fluid dynamics design is developed, 
structural integrity becomes the primary concern. Until that integrity can be ensured, the 
design is reworked. 

Flow-field knowledge is acquired through experimentation in ground-based test facilities 
and by theoretical methods; “measurable” 2 technology and theoretical technology may have 
to be developed in order to acquire the flow-field information that is needed. Unlike aircraft 
such as the X-15, SR-71, and F-16, for which old-time slide rules and desk calculators 
provided most of the theoretical fluid-dynamics design input, the aerospace plane design will 
rely heavily on supercomputers. 

In the 1970s the Space Shuttle was built largely with off-the-shelf technology. 3 Two ap- 
proaches were used in the design process: analysis and measurement. The analysis was based 
on simplifying assumptions about the governing equations, physics, and simple shapes. It 
was later determined that the ground-based testing and the design methods that were used 
did not do a good job of predicting the aerodynamic characteristics, which were actually 
determined only during the Space Shuttle flight program. 4 - 5 For example, the orbiter ex- 
perienced nose- up pitching moments, during entry at hypervelocities, that exceeded the 
established variations. 6,7 Because the Shuttle was designed using tolerances (based on mea- 
surement scatters) and variations (Rom model to full-scale) in aerodynamic coefficients, it 
was possible to tolerate very large discrepancies in preflight aerodynamic data. 5 The lessons 
learned from the Space Shuttle flight program suggest that improved design methods must be 
developed, even for designing aerospace planes that use largely state-of-the-art technologies. 
Such design methods would be based on improved ground-based measurements and on com- 
plex analyses involving the various forms of the Navier-Stokes and higher-order equations as 
well as actual vehicle shapes. 

The existing data base for designing SSTO aerospace planes with supersonic combustion 
is very limited, and the existing ground-based test facilities and test techniques are deficient 
for developing an adequate data base. 8-13 Currently planned new facilities and modifications 
of existing facilities will be a great help, but ground-based testing has its limitations. There 
are fundamental difficulties in creating complete flight simulations in ground-based facilities. 
Computational fluid dynamics, on the other hand, can go a long way toward filling the 
present void in the data base and complementing ground-based testing. Specifically, in 
major portions of the flight envelopes of these planes, CFD will be required to determine 
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the performance and specifications of the fluid dynamics designs. 

The challenge posed by all aerospace plane design efforts is to obtain credible CFD 
results, to assess the probable (quantified) uncertainties in those results, and to certify the 
codes as tools. The credibility of performance quantities is critical in the design process. 
The uncertainties lead to assessments of the success risk and safety risks associated with 
the fluid dynamics designs. The computer codes that generate credible results are certified 
by documenting their strengths and weaknesses, limitations, range of applicability and^the 
manner of their use. Further, this challenge is managed by “designing the designers” to 
properly use CFD, by instituting checks and balances in the process of generating CFD 
results, and by having a balance of computations and measurements. Fortunately, technical 
solutions for addressing this challenge are at hand. 


Credibility of CFD Results Used for Design 

The credibility of the CFD results is the responsibility of the users of those results. In 
the context of the aerospace plane, the users are the designers, the computational fluid 
dynamicists, and the measurable fluid dynamicists who are involved in the design process. 
They may either establish this credibility themselves or they may use appropriately certified 
codes. Just as credibility, or the lack of it, is a characteristic of computed results, reliability, 
or the lack of it, is a characteristic of codes. Certified codes are not available for the 
design of aerospace planes; the CFD design technology is yet to be developed (Fig. 1). 
Ultimately, the credibility of the design is determined by the credibility of the computed 
results used in the design process. Note that the use of certified codes does not necessarily 
establish the credibility of the design; a reliable code may be used improperly. Various 
aspects of the credibility issue as it regards CFD results for use in the design process as well 
as recommendations for establishing that credibility are discussed below. 

Computational and Measurable Fluid Dynamics Validation 

Some observations about the status of “CFD validation” and issues relevant to validation, 
calibration, and verification have been discussed previously. 14 - 16 Recently, CFD validation 
was declared to be a process that never achieves its objectives. “In principle, validation 
is a continuous never ending process.” 17 And, “The development process . . . presents a 
dilemma because code validation is actually an open-ended process brought about by the 
ever- widening scope and accelerated pace of possible applications compared with the ability 
to provide experimental validation and by algorithm enchancements that are progressively 
being introduced.” 18 Again, “Validation ... is an ongoing process that parallels the 
development of CFD tools.” 19 Such declarations may be appropriate in all types of research, 
except programmatic research, 14 or may be used for obtaining carte blanche to work on 
code (or CFD) validation or to continually justify new test facilities and instrumentations. 
However, there has to be a periodical statement of the status of validation based on the state 
of the art of CFD for a class of flow problems. In this sense, validation is a time-constrained, 
closed-end process. Further, the focus when using CFD results in design should be on the 
level of their credibility, or on the quantified uncertainties, and on improving that credibility 
rather than on validation. 
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Recently, the results of 38 computations from three codes were compared with measure- 
ments, or with the results of other computations, for the purpose of code calibration for 
configuration analysis. 20 That comparison study is a good starting point for CFD calibra- 
tion, but the bottom line is missing: the codes are not reported to have been calibrated. 
What can these codes predict accurately? What is the credibility of the computed results? 
What are the limitations of the computed results? When should these codes be used? How 
well can the performance quantities be predicted? Without providing answers to these kinds 
of questions, such comparisons are incomplete for application to the design process. Another 
example is a quite good status report of the validation efforts undertaken for two codes. 21 
It concludes with the following statement: “Excellent agreement [is] obtained between com- 
putations and experiment for a wide variety of test cases.” But it, too, fails to answer the 
above questions. Specifically, “excellent” needs to be quantified for design applications. 

In the aforementioned examples, it is not stated what code options, if any, were used 
when the results presented were generated. These options should be a part of the status 
report. If different options were used, then the various results generated with a code are not 
generated with the same code, even though the name of the code is the same. On the other 
hand, often a change is made in a code but not in its name. The details of the code are what 
matter, not its name. 

Computational fluid dynamics validation is no different from measurable fluid dynam- 
ics (MFD) validation related to ground-based testing for simulating flight reality (Fig. 2). 
There are uncertainties in both computed results and in measurements. The CFD validation 
requires addressing uncertainties introduced by 1) a lack of equivalence between theoreti- 
cal and computational models, 2) unsatisfactory computational accuracy, 3) isolation of 
phenomena, 4) extraneous phenomena, 5) improper modeling of phenomena, 6) creative 
overbelief, 7) definitions, 8) risk assessment, and 9) the decision-making process. The MFD 
validation requires addressing uncertainties introduced by 1) interference effects, 2) unsatis- 
factory measurement accuracy, 3) insufficiency of measurements, 4) isolation of phenomena, 
5) extraneous phenomena, 6) creative overbelief, 7) definitions, 8) risk assessment, and 9) 
the decision-making process. Isolation of phenomena and extraneous phenomena include 
the uncertainty owing to scaling from ground-based tests to flight conditions. The crucial 
differences between CFD validation and MFD validation are modeling of phenomena and 
insufficiency of measurements. (A discussion of various uncertainties in CFD and MFD for 
hypersonic flight estimates is given in Ref. 16.) In spite of the limitations of the measure- 
ments, designers have been able to design aircraft that are now flying at subsonic, transonic, 
and supersonic speeds. The limitations of computed results must be determined and then 
those results must be appropriately applied to the design of aerospace planes that will fly at 
hypersonic speeds. 

Responsibility for Credibility 

Sometimes, results computed by the originating organizations are considered more credi- 
ble than those generated by others. Because of “functional parochialism” those who conduct 
research usually do not produce results needed by the users. (“Functional parochialism” is 
a phrase attributed to Stephen Holmes by Cortes-Comerer. 22 ) In fact, workers in industry 
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are better suited for and have more incentive for and experience in design-like computations 
and design computations than do research institutions. Actually, computed results have to 
be evaluated irrespective of who generated them. 

The design CFD codes may be developed from research codes. The purpose of research 
codes is to develop and validate numerical and physical modeling methods. Establishing 
the credibility of CFD research results does not necessarily establish the credibility of CFD 
design results. Research laboratories are responsible for developing the research codes and 
for the numerical and physical modeling concepts; however, they are not responsible for 
the credibility of computed results used in the design process. On the other hand, the 
computational fluid dynamicists involved in the design process are responsible for developing 
design codes. Users of CFD are principally responsible for using CFD appropriately and for 
ascertaining the credibility of computed results. The burden of proof lies with the users 
to show that the level of complexity used when the results were generated was such that 
further complexity would not change the conclusions drawn from the results. In addition to 
this issue of the level of complexity, the burden of proof also lies with the users and code 
developers to show that CFD uncertainties have been addressed and that the credibility of 
CFD results has been established, if the codes used have not been certified. 

The primary responsibility of a CFD group in industry is to use CFD to support the 
design process by determining performance estimates and global flow fields that are accurate 
and affordable. To fulfill this responsibility, the CFD group has the following tasks: develop 
CFD design technology, participate in the design process, develop and enchance CFD tools, 
maintain quality control of CFD results, and determine probable uncertainties in the com- 
puted results. Thus, the CFD group is not only responsible for CFD methods, but for CFD 
design technology. Methods are a part of technology. 

Credibility Requirement 

The issue is not CFD validation or code validation or code calibration. The issue is 
what is the level of credibility of computed results or what are the quantified uncertainties 
associated with these results for designing the aerospace plane so that it will meet some 
specific operational goals. The designers associated with a program and oversight committees 
or committees external to this program are concerned with the credibility of computed 
results that lead to the design of a fluid-dynamics system. Recall that the Space Shuttle was 
designed with tolerances and variations. Likewise, the design and development of aerospace 
planes require margins that can be determined by quantifying uncertainties. 14 A validated 
or calibrated code is not sufficient; such a code also has uncertainties. These uncertainties 
are needed in the design process. Considering that the modeling, for example, of turbulence 
is still not in hand after more than 50 years of serious effort, CFD validation, which has been 
in vogue since the mid 1970s, is an elusive goal for current aerospace plane programs. If 
CFD validation is defined as a process , 17 “ 19 ' 21 then the outcome of that process is of doubtful 
use in a design application. What is of a paramount concern for the fluid-dynamics design is 
quantification of the CFD uncertainties and the management of those uncertainties so that 
their magnitude is appreciably reduced. 
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The fluid-dynamics specifications of a fluid-dynamics system are essentially determined 
by the performance parameters and, to some extent, by the global flow fields. These param- 
eters and flow fields are important in the design process. Therefore, their credibility must 
be determined. The CFD credibility requirement calls for the determination of the level 
of credibility or quantification of the uncertainties related to CFD-determined performance 
estimates and global flow fields. A related issue is computational fluid dynamics accuracy. 
The utility of CFD results determines the level of accuracy required of these results. In 
either case, the approach is a top-down one. 

The philosophy of utilizing the top-down approach is new when applied to the use of 
CFD in the design process. Computational requirements for making hypersonic flight perfor- 
mance estimates to implement this philosophy have been discussed previously . 14,23 Recently, 
Povinelli 24 and Marvin 25 for the first time discussed further applications of this philosophy. 
The performance estimates are required for making design evaluations, determining design 
sensitivities and optimization, establishing a design data base, and providing ground-based 
test support. The level of accuracy required is quantified by the degree to which specifica- 
tions are sensitive to performance quantities and in turn the degree to which performance 
quantities are sensitive to computational errors. These sensitivities may change as a design 
evolves from the conceptual to the preliminary to the final phases. The need for ascertaining 
the required level of accuracy is dictated by the need for keeping the computational cost 
affordable. 

The global flow fields are useful for understanding the physics and in making trade-off 
studies. The primary requirement for these flow fields is qualitative accuracy to a level 
determined by their utility. For example, such accuracy is sufficient for trade-off studies 
that seek to discern global trends. At times, quantitative accuracy is required when relative 
changes in some aspects of global flow fields are sought and when performance parameters 
depend on such global characteristics. An example of the latter is the accurate determination 
of the location of the bow shock wave at the inlet plane of an air-breathing aerospace plane so 
that the inlet performance parameters may be computed accurately. Either computational 
or fluid-dynamics uncertainties or both may exist. A shock-capturing numerical method 
without a solution- adaptive grid system may cause a problem in identifying the exact location 
of the shock wave. If the physics of flow in the nose region of the aerospace plane include 
nonequilibrium chemistry but if equilibrium chemistry is used for computing the flow field, 
there may be an uncertainty in the location of the bow shock wave at the inlet plane. Further, 
quantitative accuracy in the physics of interest is necessary for the modeling of that physics 
in simplified CFD. The sensitivity of the physics to modeling determines the level of accuracy 
required. 

In order to determine trends or relative changes, modeled numerics and physics and CFD 
input parameters, except the trend-causing parameters, must be kept the same unless there 
is an appreciable change in the physics. Specifically, only one code, not a mixture of codes, 
should be used. The bias and precision of computation must be preserved consistently when 
trends and relative changes are being determined. The conclusions of such studies may be 
verified by another code. 
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As an example, performance parameters and global flow- field requirements for the nozzle 
of an SSTO aerospace plane with supersonic combustion are considered. The performance 
parameters are thrust, moment, and heat loads. The global flow fields involve a combination 
of some of the following phenomena: turbulence, relaminarization, retransition, catalyticity, 
chemical kinetics, combustion, shock interactions, shear-layer mixing, separation, secondary 
flows, combustion- nozzle interactions, aerodynamic interactions, module-to-module interac- 
tions, unsteadiness, and three-dimensionality. Computations of performance parameters and 
global flow fields determine the requirements for CFD. The level of computational accuracy 
required in performance parameters is determined by the sensitivity of these parameters to 
the specifications for the aerospace plane, for example, the takeoff gross weight, such that the 
operational goals set for it are achieved. Further, these parameters are sensitive to physical 
models. Likewise, the utility of global flow fields determines what physical phenomena need 
to be captured and with what complexity. Again, the issue is sensitivity. Essential aspects 
of sensitivities with examples have been discussed previously. 14,16,26,27 

A Guide for Establishing Credibility 

There are two approaches to establishing the credibility of CFD results: measurable 
and computational. Since CFD numerically simulates physical and chemical reality through 
modeling, a comparison of computed results and measurements should determine the level 
of the credibility of the computed results. Essentially, the computational approach does not 
utilize test measurements. In this case, the credibility of modeling, of input parameters, of 
equivalence, and of numerical accuracy is determined. A sensitivity-uncertainty analysis is 
used. Both of these approaches are equally important and both should be used. They are 
complement ary. 

The CFD design technology development for space planes is done as follows. First, 
a systematic determination of the numerical accuracy of performance quantities is carried 
out. Second, the computed physics/chemistry is determined to be qualitatively satisfac- 
tory. Third, a part of the physics/chemistry modeled is validated by measurements and the 
fluid-dynamics uncertainties are determined, limited to this physics/chemistry found in unit, 
benchmark, or design-like flow problems. Fourth, computations that are likely to be unreli- 
able are identified, and a sensitivity-uncertainty analysis is performed. Fifth, the complete 
physics/chemistry that is modeled is evaluated and the CFD uncertainties are quantified, 
and, if the modeling is not satisfactory, it is improved with measurements in flight tests. 
Note that the CFD design technology for a class of aerospace planes can be considered de- 
veloped and ready for use as a tool only after this fifth item is addressed, and this has to be 
done before the completion of a technology development program. 

It is nearly universally observed by people who participated in research, development, 
and flight-test activities of past experimental programs that “their” program was ended 
prematurely. 28 The research and technology development program for an experimental plane 
is over soon after the plane’s flight-test program is completed. But as long as the program 
continues, the technology development must continue. One of the challenges of the X- 
30 flight-test program is considered to be “the validity of the predictions upon which the 
vehicle design will be based.” 29 Rather the challenge for successful development of NASP 
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derivative planes is to gather sufficient data so that the CFD design technology can be further 
developed while the flight-test program is under way. This continued development will also 
provide preflight computations to assist in the safe expansion of the flight envelope. 14 ’ 30 

The present situation relating to SSTO planes with supersonic combustion is no different 
from that prevailing in the computation of nuclear clouds. In the latter, there is a lack of 
good measurements. Particularly, the initial conditions required to numerically simulate the 
events are not always complete, and the background turbulence level is not known. The cur- 
rent effort consists of a four-point attack: 31 validation of first-principles sub-models against 
benchmark data; single-parameter sensitivity studies to determine important parameters, 
one-to-one comparisons of the entire model with a suite of nuclear test data; and variations 
around specified baseline simulations to determine the robustness of the simulations. This 
effort is essentially the “method-confident” approach previously 14 ’ 16 proposed and discussed 
for hypersonic flight estimates leading to code certification and to the development of cred- 
ibility in CFD results; it was outlined above for the development of CFD design technology. 

Measurements in ground-based tests or flight tests can establish the credibility of com- 
puted results if the following conditions are met: 14 1) flow quantities at the boundaries of 
the computational domain are available; 2) a relevant and sufficient quantity of qualitative 
and quantitative data are taken; 3) a data-uncertainty analysis is carried out to determine 
the bias (fixed) and precision (random and variable but deterministic) of measurements; 4) 
in design-like test articles, relevant performance quantities are measured; and 5) an indepen- 
dent evaluation of the quality of data is done. An additional condition for ground-based tests 
measurements is that the same model is tested and the same types of data are collected in at 
least two different facilities. The details of conditions 1), 2), and 4) should be determined by 
computational fluid dynamicists and by measurable fluid dynamicists before the test articles 
are fabricated. They need to function as a team. 

A code- validation task force has recommended a useful list of experiments to achieve CFD 
validation for the N ASP. 25 Some of the recommended experiments have not documented free- 
stream conditions. Without knowing the free-stream conditions, meaningful computations 
cannot be done for establishing the credibility of computational results. If these conditions 
are not known, then either measurements or computations should be done to define them. 
Computations may also be done to verify stated conditions. Further, a major shortcoming of 
this list is that the credibility of the measurements of the recommended experiments is not 
evaluated. Unless measurement uncertainties are known, the uncertainties in the computed 
results cannot be determined with these measurements. 

A statement of criteria for a successful experimental test and a detailed CFD prediction 
of how the test article is going to perform should be completed before the test. This helps 
to establish the credibility of computed results, to validate the technical understanding of a 
design, and to ensure the effectiveness of a test program. Pre-test computations also help to 
eliminate surprises during tests and to identify poor test-article designs before these articles 
are fabricated and tested. 

If measurement uncertainties are acceptable, then the uncertainties in CFD results are the 
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differences between measurements and CFD results. Of course, this definition of uncertainty 
requires that the computed results be accurate. 

The CFD results are sensitive to numerical and physical modeling factors. The numeri- 
cal factors are equivalence between theoretical and computational models and computational 
accuracy. The only way the former sensitivities (equivalence) can be determined is by com- 
paring two computed results, one without equivalence and the other with equivalence. The 
latter sensitivity (computational accuracy) is determined by grid-refinement studies. In the 
case of air-breathing SSTO space planes, the propulsion performance parameters are sensi- 
tive to global conservation of conservative variables and to the entropy condition. On the 
other hand, the physical factor is the physical model, for example, the turbulence model 
and flow-field governing equations. When measurements are not available, it is sometimes 
possible to determine sensitivity by computing the result in question with different models. 
Further, the sensitivity of the results to the input parameters is a factor that may be either 
computational or physical. These parameters include the computational model parameters 
and specifications, such as grid spacing, turbulence model constants, and the boundary con- 
ditions. A practical uncertainty analysis method for determining the uncertainties of the 
input parameters is based on sensitivity analysis . 32 

One of the computational requirements for hypervelocity flight performance estimates is 
consistency in determination of these estimates . 14 This condition requires that the designers, 
computational fluid dynamicists, and measurable fluid dynamicists use the same method for 
making these estimates. Differences in these estimates owing to differences in methods are 
not considered to be CFD uncertainties. Because of differences in the methods, different 
computational fluid dynamicists may get different estimates for the same computed flow 
field. Consistency in the method is obviously essential. 

A comparison of the results of different, uncertified codes solving the same governing 
equations and the same physical problem does not necessarily establish confidence in any 
one of these codes. Contrary to the claim that “numerical errors can be effectively identified 
by solution-to-solution comparisons .... [resulting from] code-to-code comparisons ,” 18 
these errors are unlikely to be so identified. All codes may produce similar or different 
results. And similar results may all be erroneous. On the other hand, it is not at all 
obvious from such comparisons which one of these sets of results is the correct one. Without 
knowing the magnitude of computational uncertainties, it is not possible to identify the best 
among the sets of results. In order to understand the variations between the different sets 
of results, these uncertainties need to be traced to their sources. These sources are the level 
of grid dependency (j/ + , conservation of conservative variables, etc.) and equivalence. The 
use of the computer codes should also be investigated. An improper use is not an error 
or uncertainty; it is a mistake. If there are no user mistakes in generating the results and 
if all similar sets of results have similar levels of uncertainties, then differences in these 
sets of results provide error bands owing to computational variations resulting from the use 
of different computational “wind tunnels,” just as different ground-based facilities do by 
providing slightly different measurement values for the same conditions. 

Both computational and measurable approaches for establishing the credibility of CFD 
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results are used as follows. First, the computational approach is used to determine the 
computational uncertainties. Then the measurable approach is undertaken to determine the 
uncertainties resulting from physical modeling by comparing these results with measure- 
ments that incorporate known measurement uncertainties and for the same flow conditions. 
These measurements may be related to unit, benchmark, or design-like problems or to flight 
problems. If there are no measurements available for the flows of interest, fluid-dynamics un- 
certainties of the computed results are determined. These uncertainties are essential at least 
for the modeled flow physics that has not been validated in unit, benchmark, or design-like 
problems. Those models, which have been validated, should be shown to be applicable to 
the flows of interest. Further, there are sometimes uncertainties associated even with these 
validated models. Whether these uncertainties can be transferred over from those for unit, 
benchmark, or design-like problems to the problem of interest needs to be investigated. When 
measurements are not available and the computational uncertainties are determined but the 
fluid-dynamics uncertainties are hard to determine, then a subjective process of establishing 
the credibility is carried out by submitting the results to a team of experts for evaluation. 
Only measurements related to problems of interest, such as flight problems, allow modeling 
uncertainties to be determined directly. In this sense, the CFD design technology is ready 
to use as a tool only after the flight measurements are used for establishing its credibility. 

At the beginning of a program, the available numerics and physics are evaluated based 
on the likelihood of program requirements. Considering accuracy and :.ffordability during 
the life of the program, the top two types of numerics and physics are chosen for each ap- 
plication. A CFD consistency is sought in patching solutions across different applications or 
computational zones. These numerics and physics are coded in computer codes such that 
changes and additions related to them are relatively easy to incorporate. The availability 
of computer systems should also be considered. There should be one primary code and one 
backup code for each application, both of which are put through a certification process. 
During the course of the program, the numerics and physics may change. When that hap- 
pens, a part of the process of certification or the entire process is repeated. However, at 
major milestones in the program the status of this process in terms of current levels of CFD 
uncertainties should be available to the designers. By the end of the program, there are 
CFD tools available that can be used by designers in subsequent programs dealing with the 
same class of aerospace planes. These tools may require some modifications based on latest 
knowledge, but there is no need to develop all new tools. The procedures for modifying and 
developing additional tools would have been essentially established for the next program. 

Success Risk Assessment 

The primary requirements of CFD are 1) that the physics/chemistry of the boundary 
layer, mixing, and combustion are adequately modeled, and 2) that the necessary computing 
resources are available. The inability to fulfill these requirements leads to uncertainties in 
CFD results, which in turn lead to risk. Risk is never eliminated. The pertinent questions 
about risk are as follows: What is the magnitude of the risk? How much risk is acceptable? 
How should risk be managed? For establishing the level of credibility of CFD results, the first 
question is addressed in this section and the third is, partially, addressed here and elsewhere 
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in this paper. The acceptance of risk depends on the quality of risk assessment methods, 
the rigor with which risk abatement is carried out, and rationally formulated criteria of 
acceptance. In addition to these technical considerations, the acceptable level of risk may 
be influenced by social or political judgments. 

Risk analysts define risk as a combination of the probability of the occurrence of an 
undesirable event and each and every foreseeable consequence of that event. 33 The System 
Engineering Management Guide defines risk as the uncertainty in attaining a standard. 34 The 
National Aeronautics and Space Administration (NASA) has used the following definition: 
“The chance (qualitative) of loss of personnel capability, loss of system, or damage to or loss 
of equipment or property.” 35 

It is suggested that there are two types of risk associated with aerospace planes: success 
risk and safety risk. Success risk is important during the conceptual and preliminary design 
phases; safety risk is a major concern during the final design phase and during the subsequent 
phases of an aerospace plane program. Success risk is the probability of not achieving 
the objectives of the program. Safety risks are the probability of potential failures and 
hazards associated with the space plane. Risk can be evaluated subjectively (qualitatively 
or psychologically) or objectively (quantitatively or logically). The Delphi technique of 
sampling experts is a subjective evaluation of risk. 34 This evaluation determines a degree 
of belief that events or effects will occur. Such an evaluation is appropriate during the 
conceptual design phase. A qualitative evaluation of risk may be feasible only after there is 
an experience of quantitative evaluations. On the other hand, a probabilistic risk assessment 
logically utilizes available evidence to quantify risk. Quantitative risk assessment is necessary 
during the preliminary design phase and during subsequent phases of the program. 

Quantitative risk assessment is required for credibility, success, margin, safety, reliability, 
and quality control. Addressing risk is addressing uncertainties. The credibility level of 
CFD results is determined by quantifying CFD uncertainties. A margin to be built into a 
design requires quantification of uncertainties, because the margin is a quantitative entity. 
The process through which the level of CFD credibility is established requires that: 1) 

the sensitivity of a design performance quantity (e.g., nozzle gross thrust coefficient) or 
specification (e.g., takeoff gross weight) of interest to numerical and physical parameters 
be identified; 2) the CFD uncertainties owing to these sensitivity parameters be quantified; 
and 3) the overall uncertainty in the chosen performance quantity or specification based 
on a collection of sensitivity parameters be determined. Once the overall CFD uncertainty 
is determined, the success risk or the margin required may be determined. In some cases, 
uncertainties in more than one performance quantity may be required; for example, in an 
air-breathing SSTO space plane the specific impulse depends on inlet efficiency, combustion 
efficiency, and nozzle thrust coefficient. 

Two very useful and recent studies are reported in Refs. 36 and 37. Reference 36 illus- 
trates quantification of CFD uncertainties essentially as suggested previously 16 , and Ref. 37 
demonstrates the determination of margin, given uncertainties in performance quantities and 
in a specification. Finley has presented a statistical technique for quantifying the credibility 
level in CFD drag predictions and determined the effect of this credibility on the vehicle fuel 
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fraction required to achieve orbit. 36 Cribbs has utilized a Monte Carlo procedure to compute 
the probability distribution of the maximum velocity of hypervelocity designs to determine 
the probability of success as a function of velocity. 37 

If the success risk is not acceptable, the design is changed. On the other hand, this risk 
is reduced, in part, by reducing the CFD uncertainties. The aforementioned risk assessment 
process identifies the uncertainties that are significant, thereby pointing the way to effective 
risk management. Since performance quantities are integrated quantities, this process also 
identifies details of the overall flow field that must be improved. 

Those working on manned aerospace plane programs should be cognizant of the lessons 
learned regarding risk assessment and management from the Space Shuttle Challenger ac- 
cident, from the human and organizational failures that led to that accident, and from the 
redesign of the Space Shuttle solid rocket booster. 35,38 ” 41 In order to assess risks, these pro- 
grams need to consider the probabilistic risk assessment procedures such as those developed 
for nuclear power plants 42 and that discussed by Gregory. 26,27 

Code Certification 

The development of a tool from a technology requires the identification of the strengths, 
weaknesses, limitations, and range of applicability of the technology. A product of the 
CFD design technology development triad (Fig. 1) is design CFD cocIjs. To use a code 
as a tool, the user must know the characteristics of the code and how to apply it. The 
process of establishing the reliability and limits of applicability of the code is called code 
certification, 23 and it is defined as follows: “The process of evaluating a computer code in 
terms of its logic, numerics, physics/chemistry, and the results, to ensure compliance with 
specific requirements.” 14 A few aspects of this process are presented below, preceded by some 
background information about software “validation and verification” (V&V) in the defense 
and nuclear industries. 

Since the advent of the computer, the development of reliable software (computer codes) 
that can perform its intended function has been a challenge. Because of this challenge, for 
example, between 1975 and 1989, at least 18 software studies were initiated in the United 
States by the Air Force, Department of Defense, and General Accounting Office. 43 The 
motivation of these agencies in addressing this challenge is explained by the fact that in 
fiscal 1990, software development and maintenance accounted for about 10% of the defense 
budget. 44 

The reliability of software is seriously determined in at least two industries, defense 
and nuclear, because of the criticality of performance and system safety. There are about 
250 Department of the Defense standards and policies that specify how software should 
be developed and acquired. 44 Because of the problems associated with existing software 
practices, a master plan was recently prepared to address software acquisition, life-cycle 
management, policies, standards, personnel, and technology. The V&V procedures initiated 
by the Defense Department for software controlling the operation of weapon systems have 
been the starting point for the V&V guidelines for software used in nuclear power plants. The 
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American Nuclear Society and the American Society of Mechanical engineers have prepared, 
respectively, guidelines for V&V of scientific and engineering computer codes for the nuclear 
industry 45 and quality assurance requirements for computer software for nuclear facility 
applications. 46 

The development of a software is usually done by a process referred to as a “waterfall 
model” or “software life-cycle phases.” Generally, the model passes through the following 
stages: requirements, preliminary design, detailed design, coding, integration and testing, 
installation and checkout, operation and maintenance, and retirement. The development is 
categorized either as precedented or unprecedented, the main difference between these two 
categories being the level of understanding of the requirements. 

The current development of computer codes for the CFD design process for aerospace 
planes falls into the unprecedented category (Fig. 1). At the completion of technology 
development aerospace programs, the development of computer programs for the follow-on 
programs would be categorized as precedented developments. 

The defense and nuclear industries develop software for specific requirements and un- 
dertake the V&V process, which has a beginning and an end. But research institutes of- 
ten develop CFD codes to have “the ever-widening scope and accelerated pace of possible 
applications.” 18 The futility of developing a code for a “wide range of applications” (Fig. 
1 of Ref. 18) has been discussed previously; 14 The numerical and physical models, such as 
numerical models satisfying the second law of thermodynamics and turbulence models, need 
to be developed and validated for possible wide applications, not codes. Codes are developed 
for specific applications, both for model validation by the research institutes and for design 
application by the industry. These codes must be certified; that is, these codes must go 
through a process that would assure or inform with certainty to whom it may concern that 
they generate reliable results. 

Unless the computational uncertainties are first determined and unless the fluid-dynamics 
modeling uncertainties are then quantified, a model cannot be validated. Unless the com- 
putational and fluid-dynamics uncertainties are quantified, design application cannot be 
credible. After uncertainties have been systematically documented and guidelines for using 
the code have been developed, the code can be a reliable tool. Code certification leads to 
the quantification of uncertainties and to the documentation of its utility for a specific ap- 
plication with specific requirements. Code certification also helps to qualify the user of the 
code. 

The idea of code certification has been supported by Roache. 47 He has suggested that pro- 
fessional societies may become involved in the task of certification of commercially available 
CFD codes. However, CFD aerospace design applications and the codes developed for these 
applications would most likely be proprietary in nature rather than commercially available. 
How are these proprietary codes certified? To inform with certainty the people who need to 
know, the code-originating company itself has to collect and demonstrate the evidence in a 
handbook of code certification. Such handbooks should also be prepared for commercially 
available codes. The code- certification handbooks store corporate CFD knowledge. Doc- 
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umentation to provide traceability is very important in code certification. The guidelines 
that may be used for code certification would, in principle, be the same as those for V&V 
developed by the American Nuclear Society. 45 

Designing the Designers for Using CFD 

Computational fluid dynamics may be used in the design process to achieve the following 
objectives: understanding the physics, making trade-off studies, determining design sensi- 
tivities and optimization, making design evaluations, developing the design data base, and 
supporting ground-based tests. “Aeropropulsion dynamics” designers, structural designers, 
and measurable fluid dynamicists, as well as computational fluid dynamicists use CFD in 
this design process. With CFD they can often address problems for which no design expe- 
rience, data base, or test techniques exist. These kinds of problems exist, for example, in 
aeropropulsion dynamics, which includes aerodynamics, heat transfer, aerothermodynamics, 
propulsion, and airframe-propulsion integration. In working on an SSTO aerospace plane 
with supersonic combustion, the propulsion flow-path designer has to begin his design at 
the nose of the plane along the lower surface of the forebody, proceed through the inlet, 
combustor, and internal nozzle, then along the external nozzle, and, finally, complete his 
design at the end of the plane. The designers need to be “designed,” as it were, to properly 
use CFD if they are to produce good designs. 

CFD Design Technology 

Computational fluid dynamics technology for application to aerospace plane designs is 
being developed. When its strengths, weaknesses, range of applicability, and limitations are 
known, it will become a useful design tool. A designer is effective only if he knows these 
characteristics of his design tools. Of course, some problems cannot be fixed with tools; 
knowledge and experience are also required. Given the same CFD code and the same flow 
conditions, two designers may compute two different sets of results if they are not trained 
in the proper use of the code. 

The CFD design technology is developed during the design phase and during the flight- 
test program (Fig. 1). During the design phase, the computational fluid dynamicist col- 
laborates with the designer and with the measurable fluid dynamicist who is conducting 
the ground-based tests. During the flight-test program, the computational fluid dynamicist 
further develops the CFD design technology by again collaborating with the measurable 
fluid dynamicist who is conducting the flight tests and by assisting in the flight-envelope 
expansion process. 

To quickly learn how to use CFD, the designer has to team up with the computational 
fluid dynamicist. This is essential, and will continue to be, until the characteristics of CFD 
are well known for a given class of design problems. The computational fluid dynamicist 
cannot develop CFD technology without knowing and experiencing how this technology is 
going to be used; he and its user must work together to ensure its rapid development. Never- 
theless, the computational fluid dynamicist is responsible for developing the CFD technology 
and for ensuring its proper use. The advantages of teaming up the computational fluid dy- 
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namicist and the designer and establishing quality control of CFD results used in design are 
suggested based on the practices in Japanese companies. 

U.S. Companies versus Japanese Companies 

Concerning U.S. and Japanese companies, the following observations can be made. Al- 
though the United States is substantially ahead of Japan in basic research and has the lead 
(slim though it may be) in applied research, with some exceptions Japan leads in the pro- 
duction of high-quality goods. 48 The U.S. companies generally have an edge in innovation, 
whereas the Japanese companies are usually more successful in incorporating innovations 
into quality products. There are at least three significant reasons for the Japanese successes. 
First, in the United States ideas are generally moved from one level or organization to an- 
other; in Japan, it is the people with the ideas who are usually moved from one organizational 
level to another. 15 Second, in the United States, the quality of a product has been generally 
controlled during manufacturing, by manufacturing process controls, and only recently and 
to a limited extent during product and process design; in Japan, the quality of a product 
is primarily controlled during the design process by designing both product and process for 
quality. Third, in the United States, precompetitive teamwork between companies is only 
just beginning to take place; in Japan, such teamwork has existed for many years. 

Teamwork 

In the United States, teamwork is of growing importance in many industries because 
it cuts lead time, improves quality, and, therefore, cuts costs. For example, the Ford Mo- 
tor Company discarded the traditional “over-the-transom” process of product design for 
planning the Taurus automobile and introduced a program team approach. 22 This approach 
brings all resources together and blurs the distinction between research and development 
and manufacturing, beginning at the start of a design and development program. In the 
over-the-transom process there is usually a “caste” system and a “culture” to resist change. 
People are often entrenched in their work and priorities. Supervisors are unlikely to reward 
contributions to the overall organizational objectives that are outside their domain of re- 
sponsibility. Thus, supervisors can put workers in catch-22 situations and discourage such 
efforts. Teamwork is hampered by the enforcement of organizational boundaries or by func- 
tional parochialism. In the team approach, people with the right cross-functional experience 
and attitude are required to form a multifunctional task force. 

Teamwork is also taking place between nations and between competing companies. Part- 
ners benefit from each other’s success, and results are often achieved that are better than 
those that could be achieved by either partner alone. An example is the National Aero-Space 
Plane (NASP) program for developing a precompetitive SSTO technology that incorporates 
supersonic combustion. 

The formation of the National Contractor Team for the NASP program is most advanta- 
geous for both the U.S. government and contractors and ensures the broadest possible U.S. 
base in hypersonic technology. The NASP program examplifies a new mode of operation 
that is a strategic partnership between the government and industry. Teamwork between 
the six members of this program, five industrial members and the government (consisting 
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of several government organizations), focuses on the best ideas and makes possible a more 
effective utilization of resources. The government member is a direct participant in the pro- 
gram work, and there is a totally open working relationship between the government and 
industry. An effective teaming at the program level requires effective teaming between the 
industrial members and between the government organizations. 

To obtain the aforementioned benefits of teamwork there has to be give-and-take be- 
tween people of diverse backgrounds and expertise. Just as in any organization, teamwork 
arrangements require safeguards against flawed decision-making processes that can result 
from the “group- think” 49 and the “Abilene” paradox 50 phenomena. Group-think is a mode 
of thinking of a cohesive in-group whose members strive for unanimity rather than realisti- 
cally appraised alternative courses of action. This phenomenon has three main symptoms, 
overestimation of group power and morality, close mindedness, and pressure to move toward 
uniformity. An apparent example is that “the sheer number of [Space Transportation Sys- 
tem] related boards and panels seems to produce a mindset of ‘collective responsibility. 

The Abilene paradox describes the tendency for organizations or groups to make decisions 
that may run contrary to the information they have about their problems, thus compound- 
ing the problems rather than solving them. For example, members of a group are often 
inclined to subordinate their own true feelings about a problem to what they perceive as 
a “group” preference. The inability to manage agreement is the primary symptom of this 
paradox. The decision to launch Challenger was an example. 51 To confront this paradox 
from within an organization is quite difficult and is often done at a personal price. These 
two decision-making situations are avoided essentially by establishing checks and balances, 
having each member of the team function as a critical decision maker, seeking independent 
evaluations, allowing team members to play the role of devil’s advocate, and providing a 
work environment that allows bad news to travel up through the organization. 

Ideas, Quality and Teaming 

The aforementioned advantages of transferring the people who have the ideas rather than 
just the ideas, designing for quality, and using teamwork suggest that designers need to work 
closely with computational fluid dynamicists to master the art of using CFD in the design 
process and to develop CFD design technology for applications in the design of aerospace 
planes. Computational fluid dynamicists can be used solely as consultants by designers only 
after the CFD design technology becomes a tool. The former have to be a part of the design 
process in order to make this technology a useful design tool, because a technology can 
be properly developed only when first-hand experience is gained in its intended application. 
The experience and knowledge of the computational fluid dynamicists make them responsible 
for the credibility and appropriateness of the computed results. They provide checks and 
balances and independent verification. The Total Quality Management System philosophy, 
which requires credible CFD results, their continuous improvement, and their affordability, 
is possible through the teamwork of designers and computational fluid dynamicists. 

The best approach for developing the CFD design technology for the designers is a 
combination of design, measurements (ground-based and flight), and computations. This 
approach is the CFD design technology development triad illustrated in Fig. 1. With 
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teamwork as its motto, a programmatic research 14 and technology development program 
such as the NASP program is required for developing CFD design technology for aerospace 
planes and for designing the designers to use this technology. 

Quality Control 

The American Society for Quality Control defines quality as “the totality of features 
and characteristics of a product or a service that bear on its ability to satisfy [a user’s] 
given needs.” 62 A better definition is given by Taguchi who said that quality is measured 
by “the loss imparted to the society from the time a product is shipped.” 53 According to 
this definition, the function of quality control is to discover and implement techniques that 
produce net savings to society. The final quality and cost of a manufactured product are 
determined to a large extent by the engineering designs of the product, the designs of the 
processes for its manufacture, and its manufacturing process itself. Designing for quality 
results in better quality than do attempts to control quality only in process design and 
during manufacture. Quality control begins with the tools and technology used for design 
and continues through all subsequent steps. Thus, quality control includes how CFD is used 
in the design process. 

Who is responsible for the quality control of the CFD results used in the design? The 
answer is that all who use CFD in the design process are responsible. Primarily, of course, the 
responsibility lies with the computational fluid dynamicist. He has to determine and provide 
guidelines for the level of CFD complexity and the level of computational accuracy required 
for ail CFD design applications; moreover, he has to monitor the process to ensure that these 
guidelines are followed. Whether the guidelines have been met should not be determined 
after the CFD results are used in the design, because such after-the-fact evaluation causes 
a societal loss (in Taguchi’s terms). If it is not possible to meet the guidelines, these results 
are flagged as deficient. Further, the guidelines help maintain uniformity and consistency in 
the use of CFD among the designers. 

The configuration and component design teams are generally concerned with the design 
rather than with the details of how they get CFD results. The computational fluid dynamicist 
needs to know how CFD technology and tools are used in design; what are the strengths, 
weaknesses, and limitations of the applicability of CFD; what are the problems in applying 
CFD; and whether there are disconnects between him and the various design teams and 
between the design teams themselves. He gets these kinds of information by teaming up 
with the designer and by being involved in the design process. This information helps to 
improve CFD quality and, consequently, that of the design. 

It is a common practice to have an independent validation and verification of software. 
Although it is required that this “checks and balances” practice be conducted by an orga- 
nization independent of the program management structure and the organizations involved 
in the program, it still assists in quality control when instituted within a program. One of 
the team members is tasked to carry out 1) spot- validation of the numerics and physical 
modeling suggested by other members or organizations, 2) quantitative spot- verification of 
computed results used in design by other team members, and 3) spot-verification of trends 
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and relative changes determined by other team members. The first of these three tasks is 
necessary for understanding and building confidence in the numerics and physics developed 
by others. The latter two help establish the credibility of computed results. 

When the CFD design data base is developed, the principles of designing with quality or 
robust design based on the Taguchi’s method 54 helps to determine what data base should 
be generated. In the traditional method of studying the effect of a large number of factors, 
their effects have been studied one at a time. Obviously, this is a costly approach. Grouping 
factors into columns that form a matrix called an “orthogonal array” 54 provides an efficient 
method of developing this data base. With the orthogonal array method many factors are 
studied simultaneously. 


A Balanced Design Process 

A system design is optimized by trading off the performance and specifications of different 
parts to achieve a balance. One of four principles may be used for balancing the design: 
equal performance, equal cost, equal safety, or equal effectiveness. The first two principles 
are widely used; the third is essential, for example, for manned space activities, nuclear 
power generation, and chemical processes. The fourth 55 is a geometric means of resource 
allocations as suggested by the first two principles. A balanced design process is achieved 
when the tools and technology used for the design are based on one of the fallowing principles: 
equal credibility, equal cost, or equal effectiveness. The design of the SSTO aerospace plane 
with supersonic combustion requires CFD for about 70% of its flight envelope. The resources 
allocated to CFD should be commensurate with the responsibility placed on it. Based on 
the principles of a balanced design process, the allocated computational cost of CFD should 
be at least the same as that of ground-based tests. Both CFD and MFD consist of tools and 
technologies. 

Historically, ground-based testing alone has not proved adequate for design purposes. 10 
Design problems associated with hypervelocity flight can be addressed properly only when 
ground-test data, flight-test data, and computations for a given class of aerospace planes are 
used together. 

The development of an operational aircraft or spacecraft is traditionally done with work 
breakdown structures related to the operational aspects as primary line items. Although 
this development policy is also applicable to the development of an operational aerospace 
plane, there should be equal emphasis on operations and technology in the development of an 
experimental (research and technology development) aerospace plane, during the conceptual 
and preliminary design phases. At the end of the preliminary design phase, the decision 
to proceed further on to final design, construction, and flight-test program is primarily 
contingent on the status of the technology rather than on the status of operations, when the 
latter is difficult to establish without a flight test. Further, the usual practice is to identify 
ground-based tests and flight tests as line items. But computations are no less important 

than the tests. 

The management of operational requirements and technology development requirements 
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is better done by a matrix organization rather than by two separate organizations. In 
particular, a three-dimensional matrix organization is required for developing the CFD design 
technology, with the design group on one side, the CFD group on the other, and the ground- 
based and flight-test group on the third side of the matrix. 

Concluding Remarks 

Computational fluid dynamics is an essential part of the design process for aerospace 
planes. The challenge posed by these design efforts is to obtain credible CFD results for 
reducing success risk and safety risks and for certifying computer codes to be used as tools. 
This challenge is addressed by determining the level of credibility or the probable uncer- 
tainties in computed results, by improving this credibility, by risk assessment, and by code 
certification. The credibility of a design is no better than the credibility of the technologies 
and tools used in its development. Addressing the uncertainties is addressing risks. It is 
recommended that success risk in the preliminary design phase and safety risks in the final 
design phase and subsequent phases of an aerospace plane program be evaluated quantita- 
tively. The challenge is managed by developing and applying the CFD design technology. 
This development and application is achieved by teaming the designers, computational fluid 
dynamicists, and measurable fluid dynamicists, by controlling the quality of computed re- 
sults, and by balancing the design process. The aerospace plane design challenge of working 
from credible computational fluid dynamics results can be met. 

Specifically, the following conclusions are noted: 

1) In the design of aerospace planes, CFD is as important as MFD. Because of the 
limitations of current ground-based facilities, in about the upper 70% of the flight envelopes 
of SSTO aerospace planes with supersonic combustion, CFD will be required to determine 
fluid-dynamics performance and specifications. 

2) What is of a paramount concern for fluid dynamics design is not CFD code validation 
but quantification of CFD uncertainties so that their magnitude is appreciably reduced and 
so that these uncertainties are used for designing with margin. 

3) Quantitative (probabilistic) risk assessment is required for credibility, success, margin, 
safety, and quality control. 

4) Model validation is a principal responsibility of research institutes, and CFD design 
application is a principal responsibility of industry. Both activities are carried out with CFD 
codes that must go through a process to inform with certainty (to certify) the reliability of 
the results produced by them. 

5) The designers need to be “designed” to properly use CFD if they are to produce good 
designs. 

6) The Total Quality Management System philosophy requires quality control of CFD 
that is used in the design process. 

7) The principles of balancing the design process need to be used to determine the cost 
of computations vis-a-vis ground-based tests. 
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THE CFD DESIGN TECHNOLOGY DEVELOPMENT TRIAD 



• BENCHMARK 
EXPERIMENTS 

Figure 1: The CFD design technology development triad (from Ref. 14). 
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COMPUTATIONAL FLUID, DYNAMICS VALIDATION 

VIS-A-VIS 

MEASURABLE FLUID DYNAMICS VALIDATION 
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Figure 2: Computational fluid dynamics validation vis-a-vis measurable fluid 
dynamics validation. 
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